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PREFACE 


The  work  reported  here  began  in  connection  with  the  ROSIE1  Language  Development 
Project,  which  was  funded  by  the  Information  Sciences  and  Technologies  Office  of  the  De¬ 
fense  Advanced  Research  Projects  Agency  under  the  auspices  of  The  RAND  Corporation’s 
National  Defense  Research  Institute,  a  Federally  Funded  Research  and  Development  Center 
sponsored  by  the  Office  of  the  Secretary  of  Defense.  Since  the  completion  of  the  ROSIE 
project,  this  work  wa«  continued  independently  by  the  author.  The  intention  in  publish¬ 
ing  this  Note,  as  well  as  its  companion  paper  Analysis  of  a  Table-Driven  Algorithm  for 
Fast  Context-Free  Parsing  (P-7452),  is  to  make  the  parsing  techniques  applied  in  ROSIE 
available  for  use  by  others  at  RAND  and  elsewhere. 

1  ROSIE  is  a  registered  trademark  of  The  RAND  Corporation. 
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SUMMARY 

This  Note  describes  a  parsing  system  developed  for  the  ROSIE  programming  language 
and  loosely  based  upon  Tomita’s  algorithm  for  general  context-free  parsing.  Tomita’s  algo¬ 
rithm  is  unique  in  that  it  defines  a  bottom-up  parser  that  uses  an  extended  LR  parse  table. 
While  this  algorithm  does  no  better  than  the  theoretical  0(n3)  time  bound  of  other  general 
context-free  parsing  algorithms  (where  n  is  the  length  or  the  sentence  being  parsed),  its  use 
of  parse  tables  eliminates  an  0(n2)  component  common  to  these  algorithms.  This  makes 
Tomita’s  algorithm  efficient  enough  for  practical  application. 

The  parsing  system  described  here  has  been  implemented  in  LISP  for  distribution  with 
the  ROSIE  programming  language,  which  has  a  highly  ambiguous  English-like  syntax.  An 
approach  to  disambiguation  and  code  generation  is  also  described,  as  is  an  algorithm  for 
constructing  the  extended  LR  parse  table. 
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I.  INTRODUCTION 


A  common  task  for  large  software  systems  is  the  translation  of  textual  input  into  rep¬ 
resentations  that  can  be  manipulated  programmatically.  Typically,  this  task  is  handled  by 
a  separate  “translator”  subsystem,  such  as  the  compiler  or  interpreter  of  a  programming 
language,  or  an  interactive  human  interface  or  user  front-end.  The  central  component  of  a 
translator  system  is  a  parser,  which  recognizes  whether  the  input  text  is  syntactically  correct 
and  outputs  a  tree  denoting  its  grammatical  structure.  While  there  are  many  parsing  algo¬ 
rithms,  only  a  few  are  efficient  enough  to  have  any  practical  utility  for  difficult  translation 
tasks.  Available  software  tools  for  automating  the  translation  process  are  limited  to  these 
algorithms  and  the  restricted  set  of  languages  they  recognize.  While  this  set  includes  most 
conventional  programming  languages,  it  excludes  languages  that  begin  to  approach  natural 
language  in  terms  of  expressiveness,  clarity,  and  readability.  The  example  that  sparked  this 
work  is  ROSIE  (Kipps  et  al.,  1987),  a  language  for  applications  in  artificial  intelligence  (AI) 
with  a  highly  ambiguous,  English-like  syntax. 

During  the  development  of  ROSIE  we  experimented  with  several  parsing  techniques. 
One  was  based  on  a  multi-track  algorithm  (Irons,  1971),  and  another  was  based  on  an  un¬ 
published  technique  developed  by  Ross  Quinlan.  Later,  these  approaches  were  discarded 
in  favor  of  a  new  technique  that  eventually  evolved  into  the  parsing  system  described  here. 
A  similar  technique  was  developed  independently  at  Carnegie-Mellon  University  (Tomita, 
1985a, b).  Tomita  defines  his  algorithm  as  a  variation  on  more  conventional  parsing  tech¬ 
nology,  with  the  result  that  it  is  easier  to  understand  and  explain.  We  have  since  refined 
the  design  of  ROSIE’s  parsing  system  after  the  pattern  of  Tomita’s  algorithm. 

A  detailed  presentation  of  this  parsing  system  is  provided  in  the  remainder  of  this  Note. 
Background  information  is  provided  in  the  next  three  sections:  An  overview  of  parsing 
concepts  is  presented  in  Section  II;  terminology  is  defined  in  Section  III;  and  standard  LR 
parsing  technology  and  its  limitations  are  reviewed  in  Section  IV.  An  informal  outline 
of  Tomita’s  algorithm  is  presented  in  Section  V,  with  an  annotated  example  appearing  in 
Section  VI.  The  algorithm  is  described  formally  as  a  recognizer  in  Section  VII,  while  it 
is  described  as  a  parser  that  returns  all  derivation  trees  in  Section  VIII.  A  discussion  of 
ambiguity  and  its  resolution  appears  in  Section  IX,  including  techniques  used  in  ROSIE  to 
select  the  best  derivation  tree.  The  app;oach  used  by  ROSIE  for  code  generation  given  a 
derivation  tree  is  outlined  in  Appendix  A,  and  an  algorithm  for  constructing  the  extended 
LR  parse  table  used  by  the  parser  is  presented  in  Appendix  B. 
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II.  BACKGROUND 

Parsing  techniques  have  been  actively  studied  since  the  early  days  of  computer  science. 
This  research  led  to  the  development  of  context-free  grammars  (CFGs)  (Chomsky,  195b), 
which  have  been  used  extensively  for  describing  the  syntax  of  programming  and  natural 
languages.1  Numerous  algorithms  have  been  developed  to  recognize  sentences  in  languages 
so  described.  Some  of  these  algorithms  are  general,  in  the  sense  that  they  are  applicable  to 
all  or  most  CFGs;  others  are  more  restricted  and  are  applicable  to  only  a  small  subclass  of 
CFGs  (including  the  grammars  of  most  programming  languages).  These  latter  algorithms, 
e.g.,  the  LL,  operator  precedence,  predictive,  and  LR  parsing  algorithms  t  4ho  and  Ullman, 
1972),  are  typically  more  efficient  than  the  former  because  they  take  advantage  of  inherent 
features  in  the  class  of  grammars  they  recognize.  This  efficiency  makes  them  appropriate 
for  use  in  realistic  applications;  translators  based  upon  these  algorithms  are  often  referred 
to  as  practical  parsers. 

Most  practical  parsers  analyze  the  syntax  of  their  input  in  a  single  deterministic  pass, 
without  need  of  backup.  Each  symbol  is  examined  only  once,  and,  at  the  time  it  is  examined, 
there  is  sufficient  information  available  to  make  any  necessary  parsing  decisions.  In  his 
famous  paper,  Knuth  (1965)  established  a  family  of  CFGs  known  as  LR(fc)  grammars  and 
provided  an  effective  test  to  determine,  for  a  given  positive  integer  k,  whether  a  grammar 
belonged  to  the  LR(/c)  class.  The  connection  to  practical  parsers  mentioned  above  is  that 
an  LR(k)  grammar  describes  a  language,  all  of  whose  sentences  can  be  parsed  in  a  single 
backup-free  parse,  if  at  most  k  symbols  of  look-ahead  are  available.  While  LR(fc)  grammars 
have  a  wide  coverage,  there  are  still  many  interesting  languages,  such  as  ROSIE,  for  which 
no  LR(k)  grammar  exists. 

The  primary  constraint  imposed  by  the  resGicted  algorithms  used  in  practical  parsers 
involves  the  level  of  ambiguity  they  allow.  A  language  is  ambiguous  if,  for  some  input  text, 
more  than  one  parse  tree  can  be  generated.  If  disambiguation  rules  exist  for  selecting  the 
“right"  parse  tree,  then  the  ambiguity  is  resolvable.  Resolvable  ambiguity  is  an  important 
aspect  of  a  language  The  grammar  of  a  language  gives  it  syntactic  structure;  ambiguity 
relaxes  the  rigidity  of  this  structure.  Examples  of  resolvable  ambiguities  found  in  conven¬ 
tional  programming  languages  include  arithmetic  expressions  and  if-then-else  statements: 

1  Chomsky  only  applied  CFGs  to  natural  language.  A  similar  notation  for  describing  programming 
languages  was  developed  independently  (Backus,  1959). 
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examples  found  in  natural  language  include  the  attachment  of  prepositions  and  relative 
clauses.  An  ambiguous  language  trades  syntactic  precision  for  conciseness,  informality,  and 
readability.  However,  when  the  ambiguities  are  resolvable,  then  the  language  regains  its 
precision  (at  least  at  a  mechanical  level).  When  the  disambiguation  rules  agree  with  the 
intuitive  expectations  of  users,  then  precision  is  also  regained  at  the  human  level.  Although 
some  practical  parsers,  such  as  those  produced  by  YACC  (Johnson,  1975),  permit  a  small 
degree  of  ambiguity  in  the  languages  they  recognize,  there  are  nonetheless  ambiguous  lan¬ 
guages,  like  ROSIE,  which  have  reasonable  and  understandable  disambiguation  rules  but 
which  cannot  be  recognized  by  these  parsers.  Such  languages  require  the  power  of  general 
context-free  parsing. 

The  general  context-free  parsing  algorithms,  e.g.,  Earley’s  algorithm  (Earley,  1968; 
1970)  and  the  Cocke- Younger- Kasami  algorithm  (Younger,  1967),  are  necessarily  less  effi¬ 
cient  than  the  restrictive  algorithms  because  they  must  simulate  a  multiple-path,  nonde- 
terministic  pass  over  the  input  tokens  (as  opposed  to  a  single-path,  deterministic  pass). 
While  many  of  the  general  algorithms  can  be  shown  to  theoretically  run  as  efficiently  as  the 
restricted  algorithms  on  a  large  subclass  of  CFGs,  in  terms  of  actual  run-time  performance 
they  are  still  too  slow  to  be  considered  practical.  Recently,  however,  Tomita  (1985a, b) 
introduced  a  general  context-free  parsing  algorithm  that  he  defines  as  a  variation  on  stan¬ 
dard  LR(fc)  parsing,  i.e.,  a  table-driven,  bottom-up  parsing  algorithm.  The  benefit  of  this 
approach  is  that  it  eliminates  the  nqed  to  expand  alternatives  of  a  nonterminal  at  parse 
time  (i.e.,  what  Earley  calls  the  predictor  operation).  For  Earley’s  algorithm,  the  predictor 
operation  is  one  of  two  0(n2)  components.  While  eliminating  this  operation  will  not  change 
the  upper-time  bound  of  0(n3),  it  can  make  the  difference  between  a  practical  parser  and 
an  interesting  toy. 

Although  Tomita’s  algorithm  has  been  shown  to  belong  to  the  same  complexity  class  as 
Earley’s  and  other  general  algorithms  (Kipps,  1988),  in  realistic  applications  it  is  unlikely 
to  ever  realize  worse  than  nlogn  time.  This  is  because  useful  languages  and  sentences 
normally  must  be  human-readable  as  well  as  machine-readable.  This  places  a  natural  limit 
on  the  complexity  of  the  input  a  parser  might  expect  to  receive.  Because  Tomita’s  algorithm 
takes  advantage  of  parse  tables  similar  to  those  found  in  practical  parsers,  implementations 
of  this  algorithm  have  the  potential  for  run-time  performance  that  is  comparable  to  the 
restricted  algorithms,  while  retaining  the  power  to  recognize  a  far  wider  range  of  languages. 


A  language  is  a  set  of  strings  over  a  finite  set  of  symbols.  These  symbols  are  called 
terminals  and  are  denoted  by  lowercase  letters,  e.g.,  a,  b,  c.  A  context-free  grammar  is 
used  as  a  formal  device  for  specifying  which  strings  are  in  a  language;  hereafter,  grammar 
is  used  to  mean  context-free  grammar.  Besides  terminals,  a  grammar  uses  two  other  sets 
of  symbols  called  preterminals  and  nonterminals.  Preterminals  define  the  lexical  classes 
of  a  language,  while  nonterminals  define  its  syntactic  classes;  preterminals  are  denoted  by 
lowercase  letters  preceded  by  an  asterisk,  e.g.,  *a,  *b,  *c,  and  nonterminals  by  capital 
letters,  e.g.,  A,  B,  C.  Together  the  terminals  (T),  preterminals  ( P ),  and  nonterminals  (N ) 
of  a  language  make  up  its  vocabulary  (V).  Individual  vocabulary  symbols  are  denoted  by- 
capital  letters  in  italics,  e.g.,  A,  B ,  C,  while  strings  of  zero  or  mor^  vocabulary  symbols  are 
denoted  by  Greek  letters,  e.g.,  a ,  /3,  7.  The  empty  string  is  e. 

A  grammar  consists  of  a  finite  set  of  rewrite  rules  or  productions  of  the  form 

A  —  a 

where  the  A  component  is  called  the  left-hand  side  of  the  production,  and  the  a  component 
is  called  its  right-hand  side.  The  nonterminal  that  stands  for  “sentence"  is  called  the  root 
( R )  of  the  grammar.  Productions  with  the  same  nonterminal  on  their  left-hand  side  are 
called  alternatives  of  that  nonterminal.  For  example, 


are  alternatives  of  A.  Productions  of  the  form 

A  — ►  c 

ar°  called  null  productions. 

The  rest  of  the  definitions  are  given  with  respect  to  a  particular  grammar  G,  called  the 
source  grammar.  We  write 

a  =>■  ,3 

if  3y,6, 77, A  such  that  a  =  -yAS  and  (3  =  ygb  and  A  — ►  77  is  a  production.  We  write 

o  3 

(a  derives  j3)  if  3o0,Oi,  •  ■  -,am  (m  >  0)  such  that. 

a  =  ao  =>  Q]  =>  •  •  ■  =>  om  =  13 . 

The  sequence  oq,  ■  ■  ■ ,  om  is  called  a  derivation  (of  ,/3  from  a). 
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A  sentential  form  is  a  string  a  such  that  the  root  R  a.  A  sentence  is  a  sentential 
form  consisting  entirely  of  terminal  symbols.  The  language  defined  by  a  grammar,  L(G), 
is  the  set  of  G’s  sentences.  Any  sentential  form  may  be  represented  in  at  least  one  way 
as  a  derivation  tree,  reflecting  the  steps  made  in  deriving  it  (though  not  the  order  of  the 
steps).  The  degree  of  ambiguity  of  a  sentence  is  the  number  of  its  distinct  derivation  trees. 
A  sentence  is  unambiguous  if  it  has  degree  1  of  ambiguity.  A  grammar  is  unambiguous  if 
each  of  its  sentences  is  unambiguous. 

A  recognizer  is  an  algorithm  that  takes  as  its  input  a  string  and  either  accepts  or  rejects 
it,  depending  on  whether  the  string  is  a  sentence  of  the  language  defined  by  the  source 
grammar.  A  parser  is  a  recognizer  that  outputs  the  set  of  all  legal  derivation  trees  of  a 
string  upon  acceptance. 
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IV.  REVIEW  OF  LR  PARSING 


Later  sections  of  this  Note  assume  familiarity  with  standard  LR  parsing,  the  exact 
definition  and  operation  of  which  can  be  found  in  Aho  and  Ullman  (1972).  In  this  section, 
we  will  briefly  review  these  concepts  by  examining  a  generic  LR  parser  as  a  recognizer.  This 
discussion  will  also  highlight  the  inherent  limitations  of  standard  LR  parsing. 

LR  parsers  derive  their  name  from  the  fact  that  they  scan  their  input  from  Left-to- 
right  and  construct  a  Rightmost  derivation  tree  in  reverse.  For  all  but  trivial  examples,  LR 
parsers  are  too  complex  to  implement  by  hand.  Their  implementation  is  straightforward  to 
automate,  giving  rise  to  a  class  of  software  tools  called  LR  parser  generators.  Such  tools 
take  a  context-free  grammar  as  input  and  output  an  LR  parser  for  the  language  described. 

LR  parsers  can  be  thought  of  as  consisting  of  two  parts:  a  driver  routine,  and  a  parse 
table.  Typically,  the  parser  produced  by  an  LR  parser  generator  will  always  contain  the 
same  driver  routine;  only  the  parse  table  changes.  In  this  way,  LR  parser  generators  are 
actually  parse  table  generators.  (A  discussion  of  parse  table  construction  is  not  important 
to  understanding  the  operation  of  the  driver  routine,  which  is  the  focus  of  this  section. 
Parse  table  construction  is  discussed  later  in  Appendix  B.) 

Figure  4.1  depicts  a  generic  LR  parser.  It  has  a  input  string,  a  stack,  and  a  parse  table. 
The  input  string  is  scanned  from  left-to-right,  one  token  at  a  time.  Each  stack  element 
records  a  symbol  s,  called  a  parse  state  (or  state).1  The  mth  element,  labeled  sm,  is  at  the 
top-of-stack  and  is  referred  to  as  the  state  of  the  parser. 


Stack 


Input  String 


Fig.  4.1 — LR  parser 


Stack  elements  also  record  useful  information,  such  as  the  sequence  of  d<  rivations  made  in  reaching  a 
parse  state.  Since  we  are  only  considering  an  LR  parser  as  a  recognizer,  this  other  information  can  be  safely- 
ignored. 
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The  parse  table  consists  of  two  parts:  an  action  table  and  a  goto  table ,  which  are 
implemented  as  the  look-up  functions  ACTION  and  GOTO  The  action  table  maps  states 
and  tokens2  to  parse  actions.  ACTION(s,x)  takes  a  state  s  and  input  token  x  and  returns 
one  of  four  actions: 

1.  Shift  to  State  s' . 

2.  Reduce  using  Production  p. 

3.  Accept. 

4.  Error. 

The  goto  table  maps  states  and  nonterminal  symbols  to  states.  GOTO(s,D)  takes  a  state 
s  and  nonterminal  symbol  D  and  returns  a  state  s' ,  which  corresponds  to  the  action  ‘Shift 
to  State  s'.’ 

Each  move  of  the  driver  routine  is  determined  by  consulting  the  action  table  entry  for 
sm  (the  current  state,  i.e.,  on  the  top  of  the  stack)  and  Xj  (the  current  input  token).  The 
moves  resulting  from  the  four  types  of  actions  are  as  follows: 

1.  If  ACTION(sm,Xi)  =  ‘Shift  to  State  s',’  the  driver  executes  a  shift  move  by  creating 
a  new  top-of-stack  element  for  s'.  Afterwards,  input  scan  is  advanced  to  Xj+j, 
making  it  the  new  current  input  token. 

2.  If  ACTION(sm,x, )  =  ‘Reduce  using  Production  p,’  the  process  executes  a  reduce 
move.  A  reduce  move  corresponds  to  the  recognition  of  a  rightmost  derivation  of 
Production  p.  If  Production  p  has  the  form 

Dp  *  Cpi  •  •  ■  Cpp 

where  p  is  the  length  of  its  right-hand  side,  then  the  top  p  stack  elements, 
*  ‘  *  3^71,  correspond  to  Cp\  *  *  *  Cpp.  The  process  first  removes  these  p  el¬ 
ements  from  the  stack,  making  sm-p  the  new  top-of-stack.  Having  recognized  the 
nonterminal  Dp,  it  then  executes  a  shift  move  to  the  state  s'  =  GOTO (sm-p,Dp) 
by  creating  a  new  top-of-stack  element  for  s'.  Input  scan  is  not  advanced  and  the 
current  input  token  is  unchanged. 

3.  If  ACTION(sm,x,)  =  ‘Accept,’  the  driver  halts  and  returns  success. 

4.  If  ACTION(sm,Xj)  =  ‘Error,’  the  driver  executes  an  error  recovery  routine,  e.g., 
displaying  a  message  indicating  the  point  in  the  input  string  where  the  error  was 
detected. 

The  driver  operates  by  first  initializing  the  stack  with  the  start  state  of  the  parse  table 
(designated  State  0)  and  input  scan  to  the  first  token  of  the  input  string,  xi.  It  then 


2 


For  this  discussion,  we  refer  to  both  preterminals  and  terminals  as  tokens. 
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repeatedly  executes  the  moves  determined  by  the  action  table  entry  for  its  state  and  the 
current  input  token  until  the  input  string  is  accepted  or  rejected. 

As  an  example,  consider  the  simple  English-like  grammar  shown  in  Figure  4.2.  The 
nonterminals  S,  NP,  VP,  and  PP  correspond  to  four  syntactic  classes:  sentence,  noun  phrase, 
verb  phrase,  and  prepositional  phrase;  the  preterminals  *det,  *n.  *v ,  and  *prep  correspond 
to  four  lexical  classes:  determiner,  noun,  verb,  and  preposition.  Each  sentence  in  this 
language  contains  a  noun  phrase,  followed  by  a  verb  phrase,  followed  by  an  arbitrary  number 
of  prepositional  phrases. 

The  parse  table  for  the  example  grammar  is  shown  in  Figure  4.3.  Entries  in  the  action 
table  have  the  meaning: 

®  shs  —  Shift  to  State  s. 

•  rep  —  Reduce  using  Production  p. 

•  acc  —  Accept. 

•  blank  —  Error. 

Entries  s  in  the  goto  table  mean  ‘Shift  to  State  s.’ 

(1)  S  ->  NP  VP 

(2)  S  —  S  PP 

(3)  NP  —  *n 

(4)  NP  — ►  *det  *n 

(5)  PP  — *  *prep  NP 

(6)  VP  ->  * v  NP 

Fig.  4.2— Example  LR  grammar 


State 

Action  Table 

*det  *n  *v  *prep  -4 

Goto  Table 

NP  PP  VP  S 

0 

sh3  sh5 

8  1 

1 

sh2  acc 

7 

2 

sh3  sh5 

6 

3 

sh4 

4 

re4  re4  re4 

5 

re3  re3  re3 

6 

re5  re5 

7 

re2  re2 

8 

sh9 

11 

9 

sh3  sh5 

10 

10 

re6  re6 

11 

rel  rel 

Fig.  4.3 — LR  parse  table 
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Next  consider  the  moves  made  by  the  driver  routine  in  parsing  the  input  ‘I  saw  the 
man’;  assume  that  I  and  man  are  equivalent  to  +n,  saw  to  *v,  and  the  to  *det.  Figure  4.4 
traces  the  nine  moves  made  by  the  driver.  (The  n  -f  1st  input  token  ‘d’  denotes  end-of- 
sentence.) 

The  column  labeled  ‘Stack’  shows  the  contents  of  the  stack  (states)  at  the  beginning 
of  each  move;  top-of-stack  is  the  rightmost  state.  Under  the  column  labeled  ‘Input,’  a  dot 
‘o’  prefixes  the  current  input  token.  For  illustrative  purposes,  when  a  segment  of  the  input 
has  been  recognized  in  a  derivation  of  some  nonterminal  (i.e.,  after  a  reduce  move),  it  is 
replaced  by  that  nonterminal.  The  column  labeled  ‘Action’  shows  the  action  table  entry  for 
each  move;  for  reduce  moves,  the  subsequent  goto  table  entry  is  also  shown. 

In  [1],  the  parser  is  in  State  0  scanning  the  first  input  token  I.  ACTION(0,*n)  =  sh5,  so 
the  driver  executes  a  shift  move,  making  State  5  the  current  state,  and  input  scan  advances 
to  saw.  In  [2],  the  driver  is  in  State  5  scanning  saw.  ACTION(5,*v)  =  re3,  so  the  driver 
executes  a  reduce  move.  The  length  of  the  right-hand  side  of  Production  3,  NP  — ►  *n, 
equals  one,  so  one  element  is  removed  from  the  stack,  making  State  0  the  current  state. 
The  parser  then  consults  the  goto  table  entry  for  State  0  and  NP,  the  left-hand  side  of  the 
production.  GOTO(0,NP)  =  8,  so  the  driver  executes  a  shift  move  to  State  8.  In  [3],  the 
current  state  is  State  8  and  I  has  been  replaced  by  NP,  but  the  input  scan  is  still  at  saw. 
Moves  [3]  through  [9]  are  determined  in  a  manner  similar  to  [1]  and  [2]. 

The  principal  limitation  of  LR  parsers  comes  from  the  fact  that  they  only  recognize 
a  restricted  set  of  context-free  grammars.  The  general  restriction  is  that  the  grammars 
must  be  unambiguous  (or  unambiguous  with  ^-symbol  look-ahead).3  Most  conventional 
programming  languages  can  be  described  by  an  LR(fc)  grammar.  However,  context-free 
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0  8 
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0  8  9  3 
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sh4 
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0  8  9  3  4 

NP  saw 

the 

man 

od 

re4 

GOTO(9,NP)  =  10 

7 

0  8  9  10 

NP 

saw 

NP 

od 

re6 

GOTO(8,VP)  =  11 

8 

0  8  11 

NP 

VP 

od 

rel 

GOTO(0,S)  =  1 

9 

0  1 

S 

od 

acc 

Fig.  4.4 — Moves  of  an  LR  parser 


3 


Although  some  LR  parser  generators  allow  a  small  degree  of  ambiguity  to  exist. 
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languages  are  inherently  ambiguous  (Ginsburg  and  Ullian,  1966),  which  means  that  there 
will  be  interesting  and  desirable  languages  for  which  no  LR(fc)  grammar  exists  for  any  k. 

To  illustrate,  suppose  we  wished  to  enhance  the  grammar  in  Figure  4.2  to  allow  a  noun 
phrase  to  consist  of  an  arbitrary  number  of  prepositional  phrases.  We  can  do  this  by  adding 
the  production 

NP  -+  NP  PP 

As  we  will  see  in  Section  5,  this  simple  change  makes  the  grammar  non-LR(£).  What 
this  means  to  our  example  is  that  the  action  table  will  contain  two  double-action  entries, 
denoting  the  two  uses  of  a  prepositional  phrase.  Since  the  driver  is  a  deterministic  process 
with  only  a  single  stack,  it  can  follow  one  of  these  actions  but  not  both.  To  follow  both, 
the  drive  must  become  nondeterministic,  using  some  form  of  search  to  parse  the  input.  In 
fact,  this  is  the  essence  of  Tomita’s  algorithm,  as  the  remainder  of  this  Note  explains. 
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V.  TOMJTA'S  ALGORITHM 


The  following  is  an  informal  description  of  Tomita’s  algorithm  as  a  recognizer.  (Fa¬ 
miliarity  with  standard  LR  parsing  is  assumed.)  Tomita  views  his  algorithm  as  a  variation 
on  standard  LR  parsing.  The  algorithm  takes  a  shift-reduce  approach,  using  an  extended 
LR  parse  table  to  guide  its  actions.  The  change  the  algorithm  makes  to  the  parse  table 
is  to  allow  it  to  contain  multiple  actions  per  entry,  i.e.,  at  most  one  shift  action  or  accept 
action  and  any  number  of  reduce  actions.  Thus,  the  parse  table  can  no  longer  be  used  for 
strictly  deterministic  parsing;  some  search  must  be  done.  The  algorithm  emulates  a  non- 
deterministic  parse  with  pseudo-parallelism.  It  scans  an  input  string  xj  •  •  •  xn  from  left  to 
right,  following  all  paths  in  a  breath-first  manner  and  merging  like  subpaths  when  possible 
to  avoid  redundant  computations. 

An  example  non-LR  grammar  is  shown  in  Figure  5.1.  This  is  a  simplistic  subset  of 
English,  where  the  nonterminals  S,  NP,  VP,  and  PP  define  the  syntactic  classes  sentence, 
noun  phrase,  verb  phrase,  and  prepositional  phrase;  and  the  preterminals  ♦det,  +n,  *v,  and 
♦prep  define  the  lexical  classes  determiner,  noun,  verb,  and  preposition.  The  parse  table 
for  this  grammar  is  shown  in  Figure  5.2. 

In  building  the  parser  table  the  grammar  i=  augmented  by  a  Oth  production 

D0  -  R-\ 

where  R  is  the  root  of  the  grammar  and  where  the  symbol  ‘T  is  a  special  terminal  that 
denotes  end-of-sentence  and  appears  only  as  the  last  symbol  of  an  input  string.  Entries  shs 
in  the  action  table  indicate  the  action  ‘Shift  to  State  s,’  and  entries  rep  indicate  the  action 
‘Reduce  constituents  on  the  stack  according  to  Production  p.'  The  entry  acc  indicates 
‘Accept,’  and  blanks  indicate  ‘Error.’  Entries  s  in  the  goto  table  indicate  the  action  ‘After 
a  reduce  action,  shift  to  State  s.’  In  Figure  5.2,  there  are  two  multi-action  entries  in  States 
3  and  11  under  the  column  labeled  *prep. 


(1) 

S 

NP  VP 

(2) 

S 

->• 

S  PP 

(3) 

NP 

-> 

♦n 

(4) 

NP 

— ► 

♦det  *n 

(5) 

NP 

— ► 

NP  PP 

(6) 

PP 

— ► 

♦prep  NP 

(7) 

VP 

-> 

♦v  NP 

Fig.  5.1 — Example  non-LR  grammar 
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State 

Action  Table 

*det  *n  *v  *prep  H 

Goto  Table 

NP  PP  VP  S 

0 

sh5  sh7 

9  1 

1 

sh2  acc 

8 

2 

sh5  sh7 

3 

3 

re6  re6,sh2  reo 

4 

4 

re5  re5  re5 

5 

sh6 

6 

re4  re4  re4 

7 

re3  re3  re3 

8 

re2  re2 

9 

shlO  sh2 

4  12 

10 

sh5  sh7 

11 

11 

re7,sh2  re7 

4 

12 

rel  rel 

Fig.  5.2 — LR  parse  table  with  multiple  entries 

The  algorithm  operates  by  maintaining  a  number  of  parsing  processes  in  parallel.  Each 
process  has  a  stack,  scans  the  input  string  from  left-to-right,  and,  thus,  behaves  basically 
the  same  as  the  single  parsing  process  in  standard  LR  parsing.  Each  stack  element  is  labeled 
with  a  parse  state  and  points  to  its  parent,  i.e.,  the  previous  element  on  a  process’s  stack. 
We  call  the  top-of-stack  the  current  state  of  a  process. 

Actually,  each  process  does  not  maintain  its  own  separate  stack.  Rather,  these  “mul¬ 
tiple”  stacks  are  represented  using  a  single  directed  acyclic  (but  reentrant)  graph  called  a 
graph-structured  stack ,  an  example  of  which  is  shown  in  Figure  5.3.1 

Each  stack  element  corresponds  to  a  vertex  of  the  graph.  (In  Figure  5.3,  circles  represent 
the  vertices  of  the  graph,  where  each  circle  is  labeled  with  a  parse  .slate.)  Each  leaf  of  the 

I  sas  the  nan  in  the  park  with  a  scope  d 

u0  (/,  U-2  r,  i i u6  u7  t‘B  tr9  i7l0 


Fig.  5.3 — Example  graph-structured  stack 


1 


Figure  adapted  from  examples  given  in  Section  VI. 
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graph  acts  as  a  distinct  top-of-stack  to  a  process.  (There  are  three  processes  depicted  in 
Figure  5.3:  one  in  State  4,  one  in  State  2,  and  one  in  State  8.)  The  root  of  the  graph 
acts  as  a  common  bottom-of-stack.  The  edge  between  a  vertex  and  its  parent  is  directed 
toward  the  parent.  (As  a  convenience  to  the  reader,  edges  in  Figure  5.3  are  labeled  when 
they  denote  the  recognition  of  a  nonterminal.)  Because  of  the  reentrant  nature  of  the  graph 
(as  explained  below),  a  vertex  may  have  more  than  one  parent.  (In  Figure  5.3,  the  vertex 
under  Us  has  two  parent  vertices.) 

The  leaves  of  the  graph  grow  in  stages.  Each  stage  U ,  corresponds  to  the  ith  symbol 
x,  from  the  input  string.  After  x,  is  scanned,  the  leaves  in  stage  Ux  are  in  a  one-to- 
one  correspondence  with  the  algorithm’s  active  processes,  where  each  process  references  a 
distinct  leaf  of  the  graph  and  treats  that  leaf  as  its  current  state.  Upon  scanning  x,+i ,  an 
active  process  can  either  (1)  add  an  additional  leaf  to  (/,,  or  (2)  add  a  leaf  to  Ui+\.  Only 
processes  that  have  added  leaves  to  U;+\  will  be  active  when  x,+2  is  scanned.  (In  Figure 
5.3,  x,  =  with;  two  leaves  have  been  added  to  Ux  and  one  to  Ui+\.) 

In  general,  a  process  behaves  in  the  following  manner.  On  x,,  each  active  process 
(corresponding  to  the  leaves  in  Ux_\ )  executes  the  entries  in  the  action  table  for  x,  given  its 
current  state.  When  a  process  encounters  multiple  actions,  it  splits  into  several  processes 
(one  for  each  action),  each  sharing  a  common  top-of-stack.  When  a  process  encounters  an 
error  entry,  the  process  is  discarded  (i.e.,  its  top-of-stack  vertex  sprouts  no  leaves  into  Ut 
by  way  of  that  process).  All  processes  are  synchronized,  scanning  the  same  symbol  at  the 
same  time.  After  a  process  shifts  on  x,  into  Ux,  it  waits  until  there  are  no  other  processes 
that  can  act  on  x,  before  scanning  x,+]. 

The  Shift  Action.  A  process  (with  top-of-stack  vertex  v)  shifts  on  x,  from  its  current 
state  s  to  some  successor  state  s'  by 

(1)  creating  a  new  leaf  v'  in  {/,  labeled  s'; 

(2)  placing  an  edge  from  v'  to  its  top-of-stack  v  (directed  towards  r>);  and 

(3)  making  v'  its  new  top-of-stack  vertex  (in  this  way  changing  its  current  state). 

Any  successive  process  shifting  to  the  same  state  s'  in  U,  is  merged  with  the  existing  process 
to  form  a  single  process  whose  top-of-stack  vertex  has  multiple  parents,  i.e.,  by  placing  an 
additional  edge  from  the  top-of-stack  vertex  of  the  existing  process  in  U,  to  the  top-of-stack 
vertex  of  the  shifting  process.  The  merge  is  done  because,  individually,  these  processes 
would  behave  in  exactly  the  same  manner  until  a  reduce  action  removed  the  vertices  labeled 
s'  from  their  stacks.  Thus,  merging  avoids  redundant  computation.  Merging  also  ensures 
that  each  leaf  in  any  Ux  will  be  labeled  with  a  distinct  parse  state,  which  puts  a  finite 
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upper-bound  on  the  possible  number  of  active  processes  and,  thus,  limits  the  size  of  the 

graph-structured  stack. 

The  Reduce  Action.  A  process  executes  a  reduce  action  on  a  production  p  by  following 
the  chain  of  parent  links  down  from  its  top-of-stack  vertex  v  to  the  ancestor  vertex  from 
which  the  process  began  scanning  for  p  earlier,  essentially  “popping”  intervening  vertices  off 
its  stack.  Since  merging  means  a  vertex  can  have  multiple  parents,  the  reduce  operation  can 
lead  back  to  multiple  ancestors.  When  this  happens,  the  process  is  again  split  into  separate 
processes  (one  for  each  ancestor).  The  ancestors  will  correspond  to  the  set  of  vertices  at  a 
distance  p  from  v ,  where  p  equals  the  number  of  symbols  in  the  right-hand  side  of  the  pth 
production.  Once  reduced  to  an  ancestor,  a  process  shifts  to  the  state  s'  indicated  in  the 
goto  table  for  Dp  (the  nonterminal  on  the  left-hand  side  of  the  />th  production)  given  the 
ancestor’s  state.  A  process  shifts  on  a  nonterminal  much  as  it  does  a  terminal,  with  the 
exception  that  the  new  leaf  is  added  to  f/,_j  rather  than  Ux.  (A  process  can  only  enter  Ux 
by  shifting  on  x^.) 

The  algorithm  begins  with  a  single  initial  process  whose  top-of-stack  vertex  is  the  root 
of  the  graph-structured  stack.  It  then  follows  the  general  procedure  outlined  above  for  each 
symbol  in  the  input  string,  continuing  until  there  are  either  no  leaves  added  to  Ut  (i.e., 
no  more  active  processes ),  which  denotes  rejection,  or  a  process  executes  the  accept  action 
on  scanning  the  n  +  1st  input  symbol  H,’  which  denotes  acceptance.  (Figure  5.3  shows  an 
instance  of  a  graph-structured  stack  in  which  three  vertices  were  added  to  f/7  as  a  result  of 
reduce  actions  while  scanning  with;  a  vertex  was  added  to  f'g  as  a  result  of  a  shift  action 
on  with  in  State  3.) 
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VI.  EXAMPLE  RECOGNITION 


To  illustrate  the  mechanics  of  Tomita's  algorithm  as  a  recognizer,  consider  the  following 
example,  which  uses  the  grammar  in  Figure  5.1  and  parse  table  in  Figure  5.2  to  analyze  the 
sentence 

‘I  saw  the  man  in  the  park  with  a  scope  H’ 

Assume  that  the  tokens  I.  man,  park,  and  scope  belong  to  the  lexical  class  *n,  that  saw 
belongs  to  *v,  that  in  and  with  belong  to  *prep,  and  that  a  and  the  belong  to  *det. 

The  diagrams  shown  below  trace  the  growth  of  the  graph-structured  stack  from  Uq 
through  U\q.  Each  circle  is  a  vertex  of  the  graph  labeled  by  a  parse  state.  The  line 
segments  connecting  vertices  are  edges  created  by  shift  actions  (i.e.,  from  the  state  of  the 
left  vertex,  on  the  grammar  symbol  labeling  the  segment,  and  into  the  state  of  the  right 
vertex).  Backward  arrows  indicate  a  reduce  action  back  to  the  vertex  at  the  head  of  the 
arrow.  Each  arrow  is  labeled  Pt.  indicating  that  an  instance  of  the  ith  production  was 
recognized.  After  reducing  back  to  the  earlier  vertex,  a  shift  is  done  on  nonterminal  D,  (the 
left-hand  side  of  the  ith  production). 

In  the  beginning,  there  is  only  a  single  active  process,  denoted  by  a  single  vertex  in  f'o 
labeled  .Fq.  The  only  action  from  State  0  on  I  is  sh7.  so  in  [l]  the  process  creates  a  new 
vertex  labeled  5 7.  adding  it  to  U\.  The  only  action  in  State  7  on  saw  is  re3,  so  the  process 
reduces  back  one  vertex  (the  length  of  Production  3)  as  indicated  by  the  backward  arrow. 
From  State  0  [2],  the  process  shifts  to  State  9  on  NP,  State  10  on  saw.  State  5  on  the,  and 
State  6  on  man.  From  State  6,  it  reduces  back  two  vertices  using  Production  4. 


I  san  the  man  in  the  park  with  a  scope  d 

Vo  V\  U,  U,  Us  Ut  U7  U,  U ,  Ut0 

scanning  x3  ■  saw  (*v) 


P, 


scanning  x,  »  in  (*prep) 


I  sa 


e  man 


Vo  V \ 

scanning  x„  »  the  (*det) 


V3  Ut 


scanning  x,  ”  the  (*det) 


scanning  x,  »  with  (tprep) 


scanning  x,  *  with  (*prep) 


From  State  10  [3],  the  process  shifts  to  State  11  on  NP,  where  it  finds  two  actions  on 
in,  sh2  and  re7.  The  process  splits  in  two.  In  [4],  one  process  follows  the  shift  action, 
while  the  second  follows  the  reduce  action.  In  [5],  the  second  process  has  shifted  on  in. 
entering  the  same  state  as  the  first  and  merging  with  it  into  a  single  process  again.  This 
merged  process  shifts  first  on  the  and  then  park  to  State  6.  where  it  reduces  on  with  using 
Production  4.  In  [fj],  the  process  encounters  another  two-action  state.  It  splits  in  two:  one 
process  follows  the  shift  action:  the  other  follows  the  reduce  action.  As  the  latter  process 
goes  hack  past  the  earlier  merged  vertex,  it  splits  once  more. 
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In  [7],  these  processes  reduce  again.  Finally  in  [8],  they  shift  on  with  to  merge  with 
the  process  that  originally  followed  the  shift  action  earlier  in  [4].  Note  that  the  process  in 
State  11  also  has  reduce  action  on  with.  It  splits  and  reduces  twice:  first  (in  [8])  using 
Production  7  and  then  (in  [9])  using  Production  1. 


[ST5T5 


scanning  xl0 


scanning  x10 


scanning  Xj 


Eventually  in  [10],  this  process  shifts  to  State  1  on  S.  Because  another  process  has 
already  made  the  same  shift,  the  recognizer  detects  an  ambiguity  and  discards  the  second 
process.  Meanwhile,  the  three  processes  that  merged  on  with  at  State  2.  shift  on  the  and 
scope  to  State  6,  and  reduce  on  H'  using  Production  1.  In  [II],  the  merged  processes 
reduce  back  past  the  merging  vertex,  where  they  split  back  into  three  separate  processes, 
two  of  which  re-merge  on  PP  in  [12]. 
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accept 


Finally  in  [13],  one  of  the  processes  reaches  a  state  with  an  accept  action  on  the  end- 
of-sentence  symbol  H,’  and  the  sentence  is  recognized  as  belonging  to  the  language. 
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VII.  THE  RECOGNIZER 


In  this  section,  a  more  precise  definition  of  the  algorithm  from  Section  V  is  presented  as 
a  recognizer  for  an  input  string  xj  •  •  •  x„.  This  definition  is  understood  to  be  with  respect 
to  an  extended  Lit  parse  table  (with  start  state  So)  constructed  from  a  source  grammar  G. 

Notation.  Number  the  productions  of  G  arbitrarily  1,  •••,</,  where  each  production  is 
of  the  form 

Dv  -r  Cv i  •  •  -Cpp  (1  <  p  <  d) 

and  where  p  is  the  number  of  symbols  on  the  right-hand  side  of  the  pth  production. 

Definition.  The  entries  of  the  extended  LR  parse  table  are  accessed  with  the  functions 
ACTIONS  and  GOTO. 

•  ACTIONTS(s,x)  returns  a  set  of  actions  from  the  action  table  along  the  row  of  state 
s  under  the  column  labeled  x.  This  „et.  will  contain  no  more  than  one  of  a  shift 
action  shs'  or  an  accept  action  acc;  it  may  contain  any  number  of  reduce  actions 
rep.  (If  the  action  set  is  empty  (which  corresponds  to  an  error),  no  actions  are  taken 
and  the  process  never  advances.) 

•  GOTO(s,D?;)  returns  a  state  s'  from  the  goto  table  along  the  row  of  state  s  under 
the  column  labeled  with  nonterminal  Dp. 

Definition.  Each  vertex  of  the  graph-structured  stack  is  a  triple  where  i  is  an 

integer  corresponding  to  the  ith  input  symbol  scanned  (i.e.,  the  time  at  which  the  vertex 
was  created  as  a  leaf),  s  is  a  parse  state  (corresponding  to  a  row  of  the  parse  table),  and 
/  is  a  set  of  parent  vertices.  The  processes  described  in  the  last  section  are  represented 
implicitly  by  the  vertices  in  successive  U{  s.  The  root  of  the  graph-structured  stack,  and 
hence  the  initial  process,  is  the  vertex  (O,So,0). 

The  Recognizer.  The  recognizer  is  a  function  of  one  argument  REC(xj  •  •  •  x„  ).  It  calls 
upon  the  functions  SIIIFT(u,5)  and  REDUCE(r.p).  SITIFT(i\.s)  either  (1)  adds  a  new  leaf 
to  U ,  labeled  with  parse  state  s  whose  parent  is  vertex  v  or  (2)  merges  vertex  v  with  the 
parents  of  an  existing  leaf.  REDUCE(u,p)  executes  a  reduce  action  from  vertex  v  using 
production  p.  REDUCE  calls  upon  the  function  ANCESTORS( c,p),  which  returns  the  set 
of  all  ancestor  vertices  a  distance  of  p  from  vertex  v.  These  functions,  which  vary  somewhat 
from  the  formal  definition  given  in  Tomita  (1985a),1  are  defined  in  Figure  7.1. 


The  changes  to  Tomita’s  algorithm  make  it  easier  to  describe  but  do  not  alter  it  significantly.  In 
particular,  Tomita’s  functions  REDUCE  and  REDUCE-E  have  been  collapsed  into  one  function. 
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REC(xi  •  •  •  x([) 

[1]  let  x„+i  :=  H 

let  b\  :=  [  ]  (0  <  i  <  n) 

[2]  let  Uq  :=  [(0,50,0)] 

[3]  for  i  from  I  to  n  +  1 

let  P  :=  [  ] 

[4]  for  Vo  =  (i  —  l.s.l)  s.t.  v  G  F',_i 

let  P  :=  P  o  [r] 

[5]  if  3‘sh  s'’  e  ACTI0NS( s,x(),  SHIFT(w ,$') 
for  V're  />’  G  ACTIONSCi ,x,)  ,  REDUCE Cw.p) 
if  ‘acc’  G  ACTI0NS(.s,x,)  ,  accept 

[6]  if  Ui  is  empty,  reject 

SHIFT (v,s) 

[7]  if  3r'  =  (i ,s,l)  s.t.  v'  e  Ui 

let  l  :=  l  U  { v } 

©X  S  Q 

'  let  Ui  :=  Ui  o  [(i..s,{r})] 

REDUCE (v,p) 

[8]  for  Vi'i'  =  (j',s',li)  i’i'  G  ANCESTORS (r, /» 

let  s"  :=  GOTO  (s' ,  Dp) 

[9]  if  3i>"  =  i’"  e  t'.-i 

[10]  if  t-i'  €  /" 

do  nothing  ( ambiguous ) 
q2.sq 

[11]  if  3v2'  =  {j',s',h')  *-t-  vo'  e  i" 

let  vc  ■■=  (i  - 

for  V‘  re  //'  €  ACTIONS  (*",  x, )  ,  REDUCE  (  vc ,  p) 


else 


[12] 

let  l " 

:=  l"  U  {V} 

[13] 

if  v" 

G  P 

let 

rc  :=  <i-l.  A 

for 

else 

V' re  p’  €  ACTI0NS(.s"  ,x,)  ,  REDUCE (:’c,p) 

[14] 

let  r,_. 

:=  o  [(,-  F,".{r/})] 

ANCESTORS (r  = 

U.-'-0,r) 

[15] 

if  c  =  0 

return({  t’}) 
else 

return(U,.'€/  ANCESTORS O'  ,c  -  D) 
Fig.  7.1 — Tlio  recognizer 
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In  REC,  [1]  adds  the  end-of-sentence  symbol  H’  to  the  end  of  the  input  string;  [2] 
initializes  the  root  of  the  graph-structured  stack;  [3]  iterates  through  the  symbols  of  the 
input  string.  On  each  symbol  x,,  [4]  processes  the  vertices  (denoting  the  active  processes) 
of  successive  U{- i’s,  adding  each  vertex  to  P  to  signify  that  it  lias  been  processed.  On  each 
vertex  v,  [5]  executes  the  shift,  reduce,  and  accept  actions  from  the  action  table  given  i.-'s 
state  5.  After  processing  the  vertices  in  C,_i,  [6]  checks  whether  a  vertex  was  added  to  lTt, 
ensuring  that  at  least  one  process  is  still  active  before  scanning  x1  +  1. 

In  SHIFT,  [7]  shifts  a  process  into  state  s  by  adding  a  vertex  to  L\  labeled  s.  If  a 
vertex  labeled  s  already  exists,  v  is  added  to  its  parents,  merging  processes;  otherwise,  a 
new  vertex  is  created  with  a  single  parent  r. 

In  REDUCE,  [8]  iterates  through  the  ancestor  vertices  a  distance  of  p  from  v.  setting 
s"  to  the  state  indicated  in  the  goto  table  under  D.,  given  the  ancestor's  state  a' .  Each 
ancestor  vertex  v\  is  shifted  into  U,-\  on  s" .  [9]  checks  whether  such  a  vertex  r"  already 
exists.  (If  not,  [14]  adds  a  vertex  labeled  to  (',_].)  If  v"  does  already  exist,  [10]  checks 
that  a  shift  from  the  current  ancestor  v\  has  not  already  been  made.  (If  it  has,  then  some 
segment  of  the  input  string  has  been  recognized  as  an  instance  of  the  same  nonterminal  Dp 
in  two  different  ways,  and  the  current  derivation  can  be  discarded  as  ambiguous;  otherwise. 
v\  is  merged  with  the  parents  of  the  existing  vertex.)  Before  merging,  [11]  checks  whether 
v\  is  a  “clone"  vertex,  created  by  [13]  in  an  earlier  call  to  REDUCE  (as  described  below). 
If  vi  is  not  a  clone,  [12]  adds  it  to  the  parents  of  v" ,  merging  processes.  [13]  checks  if  v" 
has  already  been  processed.  If  so,  then  it  missed  reductions  (if  any)  through  v\' .  To  correct 
this,  v"  is  “cloned”  into  vc  (i.e.,  a  variant  on  v"  with  a  single  parent  t>i'),  and  all  reduce 
actions  executed  on  v"  are  now  executed  on  ty. 

Returning  to  [11],  when  reducing  on  null  production,  ANCESTORS  will  return  a  clone 
vertex  as  the  ancestor  of  itself.  If  a  variant  v>'  of  r-j'  already  exists  in  the  parents  of  v” , 
then  v\  is  a  clone  and  vo'  is  the  vertex  from  which  it  was  cloned.  At  this  point  v"  has 
already  been  processed,  meaning  that  there  could  still  be  reductions  that  have  not  gone 
through  the  single  parent  of  v\  .  To  correct,  this,  v"  is  again  cloned,  and  all  reduce  actions 
executed  on  v"  are  executed  on  the  new  clone  vc- 

Finally,  in  ANCESTORS,  [15]  recursively  descends  the  chain  of  parents  of  vertex  v, 
returning  the  set  of  vertices  a  distance  of  c  from  v. 
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VIII.  THE  PARSER 


In  this  section,  the  recognizer  is  augmented  to  become  a  parser,  returning  the  set  of 
all  derivation  trees  for  the  input  string  xj  . .  .  x„  upon  acceptance.  Essentially,  two  changes 
must  be  made  to  the  algorithm.  First,  it  can  no  longer  halt  on  the  first  process  to  execute 
the  accept  action;  instead,  it  must  continue  processing  the  active  processes  until  there  are 
none  left.  Second,  each  process  must  record  derivations  as  they  are  recognized,  constructing 
portions  of  a  derivation  tree  on  the  fly.  When  the  parser  has  finished  scanning  the  input 
string,  it  returns  the  set  of  derivation  trees  recorded  by  all  accepting  processes.  Later,  a 
disambiguation  routine  can  be  applied  to  this  set  to  select  the  •‘right”  tree,  which  can  then 
be  used  for  code  generation. 

A  process  “recognizes  a  derivation”  whenever  it  executes  a  reduce  action.  Reducing  on 
Production  p  means  that  the  process  has  recognized  a  derivation  of  the  nonterminal  Dp. 
In  the  recognizer,  we  followed  the  chain  of  ancestor  vertices  to  a  distance  of  p  from  the 
top-of-stack  vertex  of  the  reducing  process.  For  the  parser,  we  can  view  each  such  chain 
having  the  form 

v\ ,  •  •  • ,  vp,  v' 

where  i’i  is  the  top-of-stack  vertex  and  v'  is  the  ancestor  that  began  scanning  for  Production 
p.  The  vertices  i'i ,  •  •  • ,  vp  correspond  to  the  symbols  Cp\  •••Cpp  in  the  right-hand  side  of 
the  pth  production,  i.e.,  the  symbols  derived  from  Dp.  If  we  assume  that  each  vertex  v, 
records  c/, ,  the  recognized  derivation  of  Cpi ,  then  when  we  shift  to  s'  (the  state  indicated  in 
the  goto  t able  for  Dp)  and  add  a  vertex  v"  labeled  s'  to  U{_\,  we  can  record  the  recognized 
derivation  of  Dp  by  attaching  the  tuple  {p,  a)  (where  a  =  [d\,---,dp])  to  the  link  between 
v"  and  its  current  parent  vertex  v' .  (This  tuple  indicates  that  the  process  has  recognized 
a  derivation  of  Dp  using  the  j> th  production  to  derive  d\  ■  ■  • dp .)  Similarly,  when  a  process 
adds  a  vertex  r  to  L ,  on  a  shift  action,  we  must  record  that  it  has  scanned  an  input  symbol 
corresponding  to  some  C'pj,  where  CPJ  is  either  a  terminal  err  pretcrminal.  To  indicate  this, 
we  attach  the  symbol  scanned  to  the  link  between  v  and  its  parent  vertex. 

In  the  recognizer,  when  the  reduce  action  causes  two  or  more  processes  to  shift  to  .s'  in 
the  processes  are  merged  into  a  single  process.  However,  when  two  or  more  processes 
shift  from  the  same  vertex  v'  to  r" ,  all  but  the  first  are  discarded  as  ambiguous.  Note  that 
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these  processes  all  recognize  a  derivation  of  the  same  nonterminal  Dp  over  the  same  input 
sequence  x;  ■  •  x,,  i.e., 

Dp  Xj  ■  ■  xt 

The  recognizer  can  discard  all  but  one  of  these  processes,  because  only  one  is  needed  in 
recognizing  the  entire  input  string.  The  parser,  on  the  other  hand,  cannot  discard  any 
of  these  processes,  because  each  recognizes  a  different  derivation  of  Dp  and  will  lead  to 
distinct  parse  trees.  So,  all  ambiguous  derivations  must  be  recorded.  To  do  this,  we  treat 
the  recognized  derivation  of  the  nonterminal  Dp  as  a  set  of  tuples  (p, a),  each  of  which 
indicates  a  competing  derivation  of  Dp  over  the  same  segment  of  the  input  string. 

The  remainder  of  this  section  gives  a  precise  definition  of  the  algorithm  as  a  parser  for 
an  input  string  xi  ■  •  •  xn .  This  definition  is  understood  to  be  with  respect  to  an  extended 
LR  parse  table  (with  start  state  So)  constructed  from  a  source  grammar  G. 

Definition.  A  recognised  derivation  is  a  set  of  tuples  (p,  a)  denoting  the  recognition 
of  a  derivation  of  Dp  using  the  pth  production  to  derive  a.  a  is  a  linear  list  of  the  form 
[r/i,  •  •  •,  dp],  where  each  dt  is  either  a  recognized  derivation  of  Cpi ,  when  C;„  is  a  nonterminal, 
or  a  symbol  scanned  from  the  input  string,  when  Cpi  is  a  terminal  or  preterminal. 

Definition.  Each  vertex  (u)  of  the  graph-structured  stack  is  a  triple  ( i,s,l ),  where  i  and 
s  are  as  before  and  where  /  is  a  set  of  tuples  (v'.d).  If  vertex  v'  reached  v  on  a  shift  action 
on  x,,  then  d  —  x,.  Otherwise,  if  vertex  v'  reached  v  from  a  reduce  action  on  Production 
p,  then  d  is  a  recognized  derivation  of  Dp. 

The  Parser.  The  parser  is  a  function  of  one  argument,  PARSE(xj  •  •  -xn).  It  finds  the 
recognized  derivation  dp  of  Xj  •  •  -x„  from  the  root  R  of  grammar  G,  returning  dp  to  denote 
the  set  of  all  derivation  trees  for  the  input  sentence.  PARSE  calls  on  the  functions  SHIFT, 
REDUCE,  and  ANCESTORS  as  defined  in  Figure  8.1. 

Figure  8.1  illustrates  the  necessary  modifications  made  to  the  functions  of  the  recognizer 
to  turn  it  into  a  parser.  In  PARSE,  [1]  adds  the  end-of-sentence  symbol  H’  to  the  end  of 
the  input  string;  [2]  initializes  the  root  of  the  graph-structured  stack;  [3]  iterates  through 
the  symbols  of  the  input  string.  On  each  symbol  x,,  [1]  processes  the  vertices  (denoting  t he 
active  processes)  of  successive  s.  adding  each  vertex  to  P  to  signify  that  it  has  been 
processed.  On  each  vertex  v,  [5]  and  [6]  execute  the  shift,  reduce,  and  accept  actions  from 
the  action  table  given  r’’s  state  s.  [6]  implements  the  accept  action  by  shifting  the  accepting 
process  into  a  dummy  accept  state  S accept •  (The  accept  action  corresponds  to  a.  shift 
on  the  end-of-sentence  symbol,  which  can  only  appear  in  recognizing  the  Oth  production, 
Do  — *  R  H.  The  recognized  derivation  d  attached  to  the  link  between  ?•  and  each  of  its 
parents  v'  is  a  recognized  derivation  of  the  root  R.)  After  processing  the  vertices  in 
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PARSE(xi • • •  x„) 

[1]  let  x„+i  :=  H 

let  Ut  :=  [  ]  (0  <  i  <  n ) 

[2]  let  Uo  :=  [(O,5o,0)] 

[3]  for  i  from  1  to  n  +  1 

let  P  :=  (  ] 

[4]  for  Vu  =  (i  — 1,5,/)  5./.  v  G  l'i-i 

let  P  :=  P  o  [v] 

[5]  if  3 * sh  6  ACTIONS (5, Xj)  ,  SHIFT(v.s') 
for  V‘ re  p>  G  ACTIONSCs.x,)  ,  REDUCED, p) 

[6]  if  ‘ace’  €  ACTI0NS(5,xi),  acrept 

[7]  if  Z/t  is  empty,  reject 

[8]  let  dji  be  [  ] 

for  Vr/  =  (?t,. s,/()  s.l.  [{n+ 1 ,  S  accept  > /)]  =  <5  (t  ,  3)  €  / 

let  f/p  :=  r/p  U  U(t)",rf'>€/' d< 

Return!  dp) 

SHIFT  (», 5) 

[9]  if  3t/  =  (*.  5,  /)  5.1.  t/  G  f 

let  /  :=  /  U  {(t,x,)} 
else 

let  t/,-  :=  Ut  o  [(/,  5,  { < r ,x() })] 

REDUCE  0, 7;) 

[10]  for  Vi'i'  =  (j',5',/1')  5.1.  €  ANCESTORS !i’,p,[  ]) 

let  s"  :=  GOTO !s(,  Dp) 

[11]  if  3i>"  =  (i-1,5",/")  5.1.  r"  G  f/,-i 

[12]  if  3 d  s.t.  (i’i \d)  G  / " 

let  d  :=  d  U  {(p,a)} 


[13] 

©  1 S  6 

if  3d,  t'_»\  //  5.1.  =  (j 7 -  -s\ 

A  (r/,d)  G  l" 

let  tc  :=  (/ -  1, 5".  {(i’i',  r/)}) 
for  V‘re  p’  G  ACTIONS!#" ,x() 

,  REDUCE !fc,p) 

[14] 

©Is© 

'  let  1"  :=  l"  U  {<r,\{p.a}}} 

[15] 

if  t’"  G  P 

«})}) 

let  rc  :=  (i  -  1 , 5".  { ( r  1 ' ,  {p, 

for  V‘ re  p’  G  ACTIONS!#",: 

c,),  REDUCE! iv, 

[16] 

else 

let  !/,_  1  :=  1  i— 1  0  [(/—  1,#",{( 

[17] 

ANCESTORS ( v  =  (j,  5.  /)  ,c,ac+i. ...p) 
if  f  =  0 

return!  {{r.  ft  1 


else 

return![J^,i  ANCESTORS ( v', c  —  1 , [f/]  o  nc  + j  ...  ?,) ) 
Fig.  8.1  The  parser 
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[7]  checks  whether  a  vertex  was  added  to  7’, ,  ensuring  that  at  least  one  process  is  still  active 
before  scanning  x;+].  After  scanning  the  n  +  1  symbol,  [8]  extracts  <7^,  the  recognized 
derivations  of  R,  from  the  single  vertex  in  returning  <7p  as  the  set  of  all  derivation 

trees  for  the  input  string. 

In  SHIFT,  [9]  shifts  a  process  into  state  s  by  adding  a  vertex  to  Ut  labeled  s  and 
recording  that  x,  was  scanned  in  reaching  this  vertex.  If  a  vertex  labeled  .s  already  exists,  r 
is  added  to  its  parents,  merging  processes;  otherwise,  a  new  vertex  is  created  with  a  single 
parent  v. 

In  REDUCE,  [10]  iterates  through  the  tuples  (v',a)  returned  by  ANCESTORS.  Each 
v'  is  an  ancestor  a  distance  of  p  from  v.  and  the  corresponding  a  is  a  list  [d\ ,  •  •  • .  df,}  of 
derivations  recorded  at  intervening  vertices.  Each  ancestor  vertex  t'i/  is  shifted  into  f',_i 
on  s" ,  the  state  indicated  by  the  goto  table  given  Dp  and  the  state  of  the  ancestor.  [11] 
checks  whether  such  a  vertex  v"  already  exists;  if  not,  [16]  adds  a  vertex  labeled  s"  to 
U,-\.  If  v"  does  already  exist,  [12]  checks  that  a.  shift  from  the  current  ancestor  v\'  has  not 
already  been  made.  (If  it  has,  then  some  segment  of  the  input  string  has  been  recognized 
as  an  instance  of  the  same  nonterminal  Dp  in  two  different  ways,  and  the  current  derivation 
{{;>,  a)}  is  merged  to  <7,  the  other  recognized  derivation  of  Dp.)  If  r x '  is  not  among  the 
parents  of  v" ,  [13]  checks  whether  it  is  a  “clone”  vertex,  created  by  [15]  in  an  earlier  call  to 
REDUCE  (as  described  in  Section  VII).  If  v\  is  a  clone,  then  there  could  still  be  reductions 
that  have  not  gone  through  its  single  parent.  To  correct  this,  v"  is  again  cloned,  and  all 
reduce  actions  executed  on  v"  are  executed  on  vc.  If  v\  is  not  a  clone,  [14]  adds  it  to  the 
parents  of  v" ,  merging  processes,  [15]  checks  if  v"  has  already  been  processed.  If  so,  then 
it  missed  reductions  (if  any)  through  v\' .  To  correct  this,  v"  is  “cloned"  into  vc.  and  all 
reduce  actions  executed  on  v"  arc  now  executed  on  the  clone. 

Finally,  in  ANCESTORS,  [17]  recursively  descends  the  chain  of  the  parents  of  vertex 
v,  collecting  the  derivations  associated  with  intervening  vertices  and  returning  the  set  of 
vertices  a  distance  of  c  from  v  and  the  list  a  of  derivations. 


IX.  DERIVATION  TREES  AND  DISAMBIGUATION 


In  Section  VI,  we  traced  the  growth  of  the  graph-structured  stack  during  the  recognition 
of  the  sentence  'I  saw  the  man  in  the  park  with  a  scope.’  The  level  of  ambiguity  for 
this  sentence  is  5  (i.e.,  it  has  5  distinct  derivation  trees,  as  shown  below  in  Figure  9.1). 
Each  internal  node  has  the  form  Dp(p),  where  Dp  is  the  nonterminal  recognized  using 


/  ^ 

/  VP(7) 

NP(3)  /  NP(4) 

I  /  /  \ 


/  ,  . 
NP(4)  /  NP(4) 

/  \  /  /  \ 


I  saw  the  man  in  the  park  with  a  scope 


S(2) 


I  saw  the  man  in  the  park  with  a  scope 


Fig.  9.1 — Five  derivation  trees 
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Production  p.  The  children  of  a  node,  ordered  from  left  to  right,  correspond  to  the  deriva¬ 
tions  of  Cpi ,  •  •  • ,  Cpp,  the  symbols  appearing  in  the  right-hand  side  of  the  pth  production. 

The  parser  outputs  these  trees  in  a  single,  compact  representation  called  a  derivation 
graph ,  as  shown  in  Figure  9.2.  In  this  representation,  boxed  nodes  denote  local  ambiguities , 
where  some  segment  of  the  input  string  can  be  derived  from  the  same  nonterminal  in  more 
than  one  way.  A  derivation  graph  has  two  advantages  over  representing  each  derivation  tree 
separately.  First,  common  subtrees  are  represented  only  once.  Second,  local  ambiguities 
are  represented  explicitly  in  a  single  node,  making  them  easy  to  detect  and  reducing  the 
amount  of  space  needed  for  their  representation. 

A  derivation  graph  can  be  input  to  a  disambiguation  function  to  extract  the  single 
“right”  derivation  tree.  For  ROSIE,  disambiguation  is  based  on  the  order  of  productions 
in  the  source  grammar,  where  productions  appearing  earlier  in  the  grammar  have  priority 
over  later  productions.  ROSIE’s  disambiguation  function  does  a.  top-down  traversal  of  the 
graph.  When  it  encounters  a  local  ambiguity,  it  compares  the  productions  used,  selects 
the  derivation  using  the  highest  priority  production,  and  discards  the  others.  It  then  con¬ 
tinues  down  the  children  of  the  selected  derivation,  looking  for  more  local  ambiguities  to 
resolve.  On  its  way  back  up,  the  disambiguation  function  pieces  the  selected  derivations 
back  together  into  a  single  derivation  tree  and  returns  that  tree  as  its  vi  ?. 


Fig.  9.2 — A  derivation  graph 
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To  illustrate,  consider  applying  this  disambiguation  strategy  to  the  graph  in  Figure  9.2. 
The  first  local  ambiguity 

rS(  1 )  S(  2  )  S(  2  T 
l _ _ I 

appears  at  the  root  of  the  graph.  The  disambiguation  function  would  select  the  derivation  of 
S(  1 )  because  Production  1  has  a  higher  priority  than  Production  2.  Discarding  1  he  other  two 
derivations  leaves  us  with  the  graph  shown  in  Figure  9.3,  which  the  disambiguation  function 
continues  to  traverse  until  it  finds  the  next  ambiguity 

i - 1 

NP(5)  NP(5) 

I _ I 

Here,  both  derivations  use  the  same  production.  To  resolve  this  ambiguity,  we  must  call  the 
disambiguation  function  recursively  to  find  the  “best”  derivation  tree  for  each  derivation 
subgraph,  and  then  do  a  left-to-right  comparison  of  these  trees,  selecting  the  tree  that  uses 
the  highest  priority  production  at  the  highest  level.  In  this  instance,  we  would  compare 
NP(5),  the  leftmost  child  of  the  first  derivation,  to  NP(4),  the  leftmost  child  of  the  second. 
Because  Production  4  has  a  higher  priority  than  Production  5,  we  would  select  the  second 
of  the  ambiguous  derivations  as  “right.”  Discarding  the  first  gives  us  the  derivation  tree 
shown  in  Figure  9.4,  which  the  disambiguation  function  eventually  returns  as  its  selection. 

A  variation  on  this  technique  that  seems  at  first  glance  to  be  better  is  to  re¬ 
solve  ambiguities  as  soon  as  they  are  detected  at  parse  time.  In  this  way,  the 


S(l) 


Fig.  9.3 — First  ambiguity  resolved 


-  30  - 


S(l) 


Fig.  9.4 — Second  ambiguity  resolved 


parser  only  outputs  a  single  derivation  tree,  thus  requiring  less  storage.  However, 
this  approach  is  not  as  time-efficient  as  the  one  outlined  above,  because  it  will 
try  to  resolve  all  ambiguities.  By  waiting  until  the  parser  is  finished,  we  avoid 
resolving  ambiguities  that  are  discarded  by  the  parser  because  they  lie  along  dead-end 
paths.  Also,  by  doing  a  top-down  disambiguation  of  the  derivation  graph,  we  avoid  resolving 
ambiguities  in  discarded  derivations.  For  instance,  we  never  had  to  resolve  the  ambiguity 

rS(l)  S12)1 
I _ I 

Finally,  by  calling  the  disambiguation  function  on  the  entire  derivation  graph,  context- 
sensitive  techniques  could  be  used,  i.e.,  contextual  information  could  be  passed  as  an  ad¬ 
ditional  parameter  to  the  disambiguation  function  and  then  used  in  extracting  the  “best" 
derivation. 

As  a  further  example,  consider  the  portion  of  a  ROSIE-like  grammar  shown  in 
Figure  9.5.  The  principal  programming  statement  in  ROSIE  is  a  rule,  which  consists  of 


(1)  <rule>  —  <block>  . 

(2)  <block>  — *  <action> 

(3)  <block>  —  <action>  AND  <block> 

(4)  <action>  —  IF  <condition>  THEN  <block> 

(5)  <action>  —  IF  <condition>  THEN  <block>  ELSE  <block> 


Fig.  9.5 — Example  ROSIE-like  grammar 
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a  block  of  one  or  more  actions  separated  by  the  reserved  word  AND  and  terminated  by  a 
period  one  type  of  action  is  an  if-then-else.  Given  this  grammar,  the  sentential  form 
action  AND  IF  condition  THEN  IF  condition  THEN  action  ELSE  action  AND  action  . 
can  have  one  of  four  interpretations,  as  shown  in  Figure  9.6.  Of  these,  Interpretation  (a) 
is  the  one  normally  found  in  modern  programming  languages,  i.e.,  actions  bind  to  the 
deepest  action  block  and  an  else  binds  to  the  deepest  if-then.  By  ordering  the  productions 
of  the  grammar  as  shown  in  Figure  9.5,  the  derivation  tree  corresponding  to  Interpretation 
(a)  will  be  returned  by  ROSIE’s  disambiguation  function. 

As  an  aside,  this  example  also  illustrates  a  situation  that,  when  recognized,  can  be  used 
to  improve  the  efficiency  of  the  parser.  Consider  the  sentential  form 

action  AND  IF  condition  THEN  action\  AND  action  AND  action 3  AND  •  •  •  actionn  . 
which  has  the  n  interpretations  shown  in  Figure  9.7. 

Again,  the  correct  interpretation  is  (a).  The  parser  makes  all  n  derivations  because, 
on  scanning  the  2nd  through  nth  AND,  it  reduces  using  Productions  2  or  3,  deriving  the 
action  block  of  the  if-then  action.  The  parser  reduces  to  a  block  on  scanning  AND  because 
an  action  can  be  followed  by  an  AND.  But,  because  a  block  can  be  the  last  component  of 
an  action,  a  block  can  be  followed  by  an  AND  as  well.  However,  we  know  tha*  with  our 
disambiguation  function  an  action  block  followed  by  an  ‘AND  action phrase  will  alwrays  be 
discarded  in  favor  of  a  longer  action  block  that  includes  that  phrase.  If  we  remove  the 
action  to  reduce  using  Productions  2  or  3  upon  scanning  an  AND,  the  parser  will  return  a 


action  AND 
IF  condition 
THEN  IF  condition 
THEN  action 
ELSE  action  AND 
action  . 

(a) 


action  AND 
IF  rondition 
THEN  IF  condition 
THEN  action 
ELSE  action  AND 

action  . 

(b) 


action  AND 
IF  condition 
THEN  IF  condition 
THEN  action 
ELSE  action  AND 
action  . 

(c) 

action  AND 
IF  condition 
THEN  IF  condition 
THEN  action 
ELSE  action  AND 
action  . 

(d) 


Fig.  9.6 — Four  interpretations 
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action 

AND 

action  AND 

IF  con 

dition 

IF  condition 

THEN 

actioni 

AND 

THEN  action ] 

AND 

actiono 

AND 

actiono 

AND 

actions 

AND 

action^  AND 

a  ctionn 

actionn  . 

(a) 

(c) 

action  AND 

action  AND 

IF  condition 

IF  condition 

THEN  action i  AND 

THEN  action i 

AND 

actiono  AND 

actiono 

AND 

actions  AND 

action^ 

AND 

a ction„  . 

actionn  . 

(b) 


(n) 


Fig.  9.7 — n  interpretations 

single,  unambiguous  derivation  tree  corresponding  to  Interpretation  (a)  and  it  will  not  have 
executed  n  —  1  spurious  reduce  actions. 

To  use  this  observation  to  optimize  ROSIE’s  parser,  we  examined  the  grammar  looking 
for  productions  that  would  generate  spurious  reduce  actions  and  then  made  declarations  to 
the  parse  table  constructor  that  would  cause  it  not  to  generate  those  actions.  Although  this 
approach  helped  to  improve  the  parser’s  run-time  performance,  it  has  not  been  generalized 
into  an  automatic  optimization  technique. 
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X.  CONCLUSION 


The  parsing  system  outlined  in  the  preceding  sections  has  gone  through  several  years 
of  evolution  since  its  initial  development  for  ROSIE.  The  parser  ciirrer.tly  being  distributed 
with  ROSIE  (Version  3.0)  (Kipps  et  al.,  1987)  is  implemented  in  Portable  Standard  LISP 
(PSL)  (Galway  et  al.,  1984),  but  is  not  up-to-date  with  the  parsing  system  described  in  this 
Note.  Essentially,  the  disambiguation  and  code  generation  functions  operate  as  described; 
the  parser  and  parse  table  constructor  are  substantially  different  in  appearance  but  are 
similar  in  much  of  their  basic  operation.  The  version  described  here  has  been  implemented 
in  Common  LISP  (Steele,  1984). 

To  make  the  advanced  parsing  technology  used  by  ROSIE  available  to  a  wider  range 
of  language  development  projects,  the  future  objective  for  this  work  is  the  development  of 
a  ‘"next-generation”  YACC-like  set  of  tools.  YACC  (Johnson,  1975)  is  a  compiler-compiler 
that  can  recognize  a  subset  of  context-free  languages  and  allows  for  a  small  degree  of  am¬ 
biguity.  YACC  is  not  capable  of  generating  a  compiler  for  ROSIE,  but  a  similar  tool  using 
Tomita’s  algorithm  would  have  reduced  the  level  of  effort  in  developing  ROSIE’s  parsing 
system  from  man-years  to  a  man-month  or  less.  In  addition,  if  this  tool  were  implemented 
in  C  as  opposed  to  LISP,  significant  performance  improvements  could  be  expected. 

As  others  begin  to  extend  the  expressiveness  and  readability  of  new  high-level  program¬ 
ming  languages,  or  as  they  try  to  improve  the  level  of  dialogue  available  in  user  front-ends, 
they  will  likewise  find  that  they  require  the  power  of  general  context-free  parsing.  Tomita’s 
algorithm  provides  a  practical  approach  to  meeting  this  need. 
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Appendix  A 

CODE  GENERATION 


For  ROSIE,  the  parsing  system  is  used  to  translate  source  programs  into  evaluable  LISP 
expressions.  Each  production  in  a  ROSIE  grammar  file  has  the  form 

( Dp  Cp\CP2  •  ■  -Cpp  ->  template ) 

where  template  is  the  component  of  the  production  that  describes  the  LISP  expression 
generated  by  it.  For  example,  a  production  in  standard  BNF  notation  (Pagan,  1981),  such 
as 

<condition>  : :=  IF  <expression>  THEN  <series>  ELSE  <series> 

would  be  described  in  the  bnf  file  as 

(<condition>  IF  <expression>  THEN  <series>  ELSE  <series> 

->  (COND  ($1  ! $2)  (T  ! $3) ) ) 

and  used  for  translating  if-then-else  statements  of  the  source  language  into  COND  state¬ 
ments  of  LISP. 

Code  Generation 

ROSIE’s  code  generation  routine  operates  by  doing  a  preorder  traversal  of  the  deriva¬ 
tion  tree  returned  by  disambiguation.  The  resulting  LISP  expression  is  generated  in  a 
left-to-right  manner  from  the  bottom  up.  At  each  internal  node  of  the  tree  (denoting  a 
derivation  based  on  Production  p)  the  routine  first  generates  the  subexpressions  for  the 
node’s  children,  i.e.,  corresponding  to  the  derivations  for  Cp\,  ■  ■  ■  ,Cpp.  If  Cp{  is  a  termi¬ 
nal,  then  no  subexpression  is  generated  for  it;  if  a  preterminal,  then  the  subexpression  is 
the  matching  input  token;  and  if  a  nonterminal,  then  the  subexpression  is  computed  by 
applying  the  code  generation  routine  to  the  corresponding  child. 

After  the  subexpressions  are  generated,  they  are  bound  to  the  variables  $  1 ,  -  -  -  ,$<7. 
respectively,  where  q  equals  the  number  of  preterminals  and  nonterminals  in  Cp\ ,  •  •  •  .Cpp. 
$j  is  bound  to  the  subexpression  generated  for  the  yth  preterminal  or  nonterminal.  Finally, 
referring  to  these  variables,  the  template  of  Production  p  specifies  the  form  of  the  LISP 
expression  to  be  generated  for  the  node. 

Template  Semantics 

The  template  of  a  production  specifies  the  expression  generated  for  the  portion  of  the 
derivation  tree  representing  an  instance  of  that  product  ion.  All  or  part  of  a  template  can  be 
used  as  a  literal;  some  portions  can  be  replaced  by  subexpressions  generated  by  nonterminals 
and  preterminals  in  the  right-hand  side  of  the  production,  while  others  can  be  replaced  by 
the  results  of  computations  specified  in  the  template. 
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For  every  variable  ( $j )  in  the  template,  the  resulting  expression  will  find  that  variable 

replaced  by  the  subexpression  bound  to  it.  If  the  variable  is  immediately  preceded  by 

an  exclamation  point  (!$i),  then  the  subexpression  must  be  a  list  and  is  spliced  into  the 

resulting  expression.  For  example,  a  list  of  one  or  more  numbers  (where  *number  is  a 

preterminal  matching  any  number)  can  be  described  by  the  two  productions 

(<numlist>  ^number  ->  ($1)) 

(<niunlist>  *number  ,  <numlist>  ->  ($1  !$2)) 

In  generating  an  expression  for  a  recognized  instance  of  the  first  production,  $1  is  bound 
to  the  subexpression  generated  for  *number,  which  in  the  case  of  a  preterminal  is  the  token 
(a  number)  matching  it.  The  expression  then  generated  is  a  list  containing  only  this  subex¬ 
pression.  In  generating  an  expression  for  the  second  production,  $1  will  again  be  bound 
to  the  subexpression  generated  for  *number,  while  $2  will  be  bound  to  the  subexpression 
generated  for  <numlist>,  which  will  be  a  list  of  one  or  more  numbers.  The  expression 
generated  by  this  production  is  a  list  containing  the  number  bound  to  $1  as  its  first  element 
with  the  remaining  elements  of  the  list  taken  from  $2 — this  is  equivalent  to  (cons  $1  $2) 
in  LISP.  Thus,  the  expression  generated  for  1,2,3  would  be  (1  2  3). 

In  order  to  allow  more  complex  semantics,  a  hook  is  provided  to  evaluate  LISP  expres¬ 
sions  at  code  generation  time  and  then  substitute  or  splice  in  the  resulting  value.  If  the 
first  element  of  a  list  in  the  template  is  the  symbol  $eval,  then  the  second  element  of  the 
list  must  be  an  expression,  which  will  be  evaluated  at  code  generation  time,  i.e., 

($eval  expression) 

This  expression  may  use  the  variables  $1 , The  value  returned  by  evaluating  the 
expression  will  replace  the  $eval  form  in  the  expression  generated  from  the  template.  If 
the  $eval  form  is  preceded  by  an  exclamation  point,  i.e., 

!($eval  expression) 

the  resulting  value  will  be  spliced  into  the  generated  expression.  For  example,  the  two 

productions  seen  earlier  could  be  specified  as 

(<numlist>  *number  ->  ($eval  (list  $1))) 

(<numlist>  *number  ,  <nuralist>  ->  ($eval  (cons  $1  $2))) 

and  generate  the  same  expression.  We  could  also  specify  the  productions 

(<sumlist>  *number  ->  $1) 

(<sumlist>  ^number  ,  <sumlist>  ->  ($eval  (+  $1  $2))) 
and  simply  generate  the  cumulative  total  of  the  numbers  in  the  list.  More  important,  $eval 
forms  can  be  used  to  check  context-sensitive  aspects  of  a  grammar  (e.g.,  a  list  of  numbers  in 
ascending  order)  or  to  make  complex  transformations  on  the  expressions  being  generated. 


-  37  - 


A  null  production  is  simply  specified  by  excluding  the  right-hand  side  of  the  production. 
For  example,  the  production 

(<sumlist>  ->  0) 

denotes  that  an  empty  sumlist  has  a  value  of  0.  If  no  template  is  specified  for  a  production, 
e-g., 

(<sumlist>  *number) 
then  a  default  template  of  the  form 
->  ($1  •••  $n) 

is  automatically  generated  for  it,  where  n  is  the  number  of  nonterminals  and  preterminals 
in  the  right-hand  side  of  the  production.  For  null  productions,  the  default  template  is  the 
empty  list,  i.e., 

->  C  ) 

If  the  template  contains  the  symbol  $tokens,  it  will  be  replaced  in  the  generated  ex¬ 
pression  by  the  list  of  tokens  recognized  by  that  instance  of  the  production.  For  example, 
given  the  production 

(<savelist>  <sumlist>  ->  ($1  $tokens)) 

the  expression  generated  for  1,2,3  would  be  (6  (1  ,2,3)).  The  symbol  !$tokens 
splices  the  list  of  tokens  into  the  expression. 
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Appendix  B 

THE  CONSTRUCTOR 

Tomita’s  parsing  algorithm  utilizes  a  minor  variation  on  standard  LR  parse  tables.  Any 
of  the  several  parse  table  construction  algorithms  (LR(0),  SLR(l),  LR(1),  LALR(l),  etc.) 
could  be  modified  to  produce  the  necessary  parse  table.  The  only  modification  required  is 
simply  to  allow  each  entry  in  the  action  table  to  record  a  set  of  actions  rather  than  a  single 
action. 

The  Extended  LR  Parse  Table 

Briefly,  to  modify  a  standard  constructor,  locate  the  portion  of  the  algorithm  that 
detects  and  deals  with  shift-reduce  conflicts  generated  when  the  algorithm  goes  to  record 
an  action  in  a  table  entry  for  which  an  action  already  exists.  Typically  when  such  conflicts 
occur,  the  constructor  either  halts  and  reports  an  error  or  records  only  one  action  and  gives 
a  warning.  For  Tomita’s  algorithm,  the  constructor  should  record  all  entries  as  a  set  of 
actions  and  add  all  conflicting  actions  to  the  set.  For  any  context-free  grammar,  each  set 
can  contain  at  most  one  shift  or  accept  action  (never  both),  and  any  number  of  reduce 
actions. 

The  parse  table  construction  algorithm  described  below  is  a  variation  on  the  SLR 
(Simple  LR)  constructor  found  in  Aho  and  Ullman  (1977).  The  central  concepts  of  this 
approach  are  (1)  to  construct  a  deterministic  finite  automata  (DFA)  that  recognizes  viable 
prefixes  of  a  grammar,  and  (2)  to  convert  the  states  and  transition  function  of  the  DFA  into 
the  states  and  actions  of  an  extended  LR  parse  table. 

Items  and  Sets  of  Items 

The  first  step  in  constructing  an  extended  LR  parse  table  is  the  construction  of  a  DFA 
that  recognizes  the  viable  prefixes  of  the  source  grammar.  A  viable  prefix  is  a  prefix  of  a 
sentential  form,  i.e.,  if  a  is  a  viable  prefix,  then 

R  a(3 

for  some  0.  A  viable  prefix  is  so  named,  because  it  is  possible  to  add  terminal  symbols  to 
the  end  of  a  viable  prefix  to  complete  a  sentential  form.  As  a  result,  we  know  that  there  is 
no  apparent  error  as  long  as  the  portion  of  the  input  string  scanned  to  a  given  point  can 
be  reduced  to  a  viable  prefix. 


-  39  - 


The  algorithm  for  constructing  the  UFA  is  couched  in  terms  of  items  and  sets  of  items. 

We  define  an  LR(0 )  item  (or  item  for  short)  of  a  grammar  G  to  be  a  production  of  G 

with  a  marker  ‘o’  appearing  somewhere  in  its  right-hand  side.  For  example,  the  production 

A  — +  XY Z  generates  the  four  items 

A  —  oXYZ 

A  -»  A'oYZ 

A  -  XYoZ 

A  —  XYZo 

Null  productions,  such  as  A  — ►  e,  generate  a  single  item,  A  — *  o. 

An  item  acts  as  a  placeholder,  indicating  how  much  of  a  production  has  been  recognized 
at  a  given  point  in  the  parsing  process.  Thus,  the  item 

A  — *•  ao/3 

denotes  the  situation  in  which  the  parser  has  scanned  input  corresponding  to  the  symbols 
in  a  and  expects  the  next  part  of  the  input  to  correspond  to  /3.  To  be  more  specific,  after 
scanning  the  ith  symbol  of  input  string  Xi  •  •  •  x„,  then,  for  some  j  (1  <  j  <  i)  and  some  m 
(i  +  1  <  m  <  n),  if 

a  Xj  x, 

then  we  expect 

0  Xi  +  1  ■■■  xm 

Items  in  which  (3  is  empty  (e.g.,  A  — *•  XYZo)  denote  the  situation  in  which  an  instance  of 
the  production  has  been  successfully  recognized  in  the  input  symbols  just  scanned. 

Where  an  item  denotes  the  partial  recognition  of  some  production,  a  set  of  items  extends 
this  concept  to  several  possible  partial  recognitions,  all  of  which  are  consistent  with  the  input 
scanned  so  far.  Individually,  items  can  be  viewed  as  the  states  of  a  nondeterministic  finite 
automata  (NFA)  for  recognizing  viable  prefixes.  Grouping  items  together  into  sets  is  similar 
to  the  process  of  constructing  a  DFA  from  the  NFA.  These  sets  of  items  later  give  rise  to 
the  states  of  the  parse  table. 

Constructing  the  Canonical  Collection  of  LR(0)  Items 

The  set  of  items  used  to  construct  the  parse  table  for  a  class  of  LR  parsers  known  as 
“simple”  LR  (SLR)  is  called  the  canonical  collection  of  LR(0)  items.  This  set  is  sufficient 
for  building  a  parse  table  to  be  utilized  by  Tomita’s  algorithm.  Its  construction  is  described 
below. 
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Notation.  Number  the  productions  of  grammar  G  arbitrarily  1  -  ,d,  where  each  pro¬ 

duction  is  of  the  form 

Dp  *  Cpi  ■  •  'Cpp  ( 1  ^  P  5~  d) 

and  where  p  is  the  number  of  symbols  in  the  right-hand  side  of  the  pth  production.  Augment 
G  with  a  0th  production 

D0  —  R  H 

where  R  is  the  root  of  G  and  ‘T  is  the  end-of-sentence  symbol.  The  purpose  of  this  new 
production  is  to  indicate  to  the  parser  when  it  has  reached  an  accept  state;  the  accept  action 
corresponds  to  shift  action  on  the  end-of-sentence  symbol. 

Definition.  A  set  of  items  (I)  gives  rise  to  successor  sets  and  transitions  between  sets 
using  the  functions  CLOSURE  and  GOTO. 

•  CLOSURE(/)  returns  a  set  of  items  constructed  from  I  by  the  following  rules. 

(1)  Every  item  in  /  is  in  CLOSURE(/). 

(2)  If  A  — *  aoB0  is  in  CLOSURE(J),  then  for  each  production  B  — *  7,  add  the  item 
B  — *  07  to  CLOSURE(7),  if  it  is  not  already  there. 

Intuitively,  A  — ►  aoBj3  in  CLOSURE(/)  indicates  that  symbols  derivable  from  Bf3 
may  be  expected  to  appear  next  in  the  input  string.  It  follows  that  for  each  pro¬ 
duction  B  — ►  7,  we  may  also  expect  to  see  symbols  derivable  from  7.  For  this 
reason,  the  items  B  — ►  07  are  added  to  CLOSURE(Z).  The  set  of  items  computed 
by  CLOSURE(J)  is  completed,  or  closed ,  when  no  new  items  can  be  added  to  it. 

•  GOTO(/,X),  where  /  is  a  closed  set  of  items  and  X  is  a  vocabulary  symbol  of  G 
(i.e.,  a  terminal,  preterminal,  or  nonterminal),  returns  the  closure  of  the  set  of  all 
items  A  — ♦  aXof3  such  that  a  subset  of  items  A  — ►  aoX/3  exists  in  I.  Intuitively, 
GOTO  establishes  links  between  closed  sets  of  items,  corresponding  to  transitions 
between  states  of  a  DFA,  i.e.,  GOTO(/,AT)  defines  a  transition  labeled  X  from  I  to 
the  set  of  items  returned  by  GOTO(/,A"). 

Each  state  of  the  LR  parse  table  is  derived  from  a  distinct  set  of  items  (I).  The  set  Iq, 
corresponding  to  the  start  state  So,  is  closed  over  the  single  item 

Do  — >  o  R  H 

denoting  that,  at  the  start  of  a  parse,  no  symbols  have  been  scanned  but  we  expect  to  see 
a  sentence  of  the  language  defined  by  G. 
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Definition.  The  canonical  collection  of  sets  of  LR(0)  items  for  a  grammar  G'  (i.e., 
grammar  G,  augmented  by  the  Oth  production  Dq  —*  R  H)  is  constructed  with  the  function 

ITEMS(G'). 

•  ITEMS^')  constructs  and  returns  C ,  the  canonical  collection  of  sets  of  items,  in 
the  following  manner. 

(1)  C  and  C'  are  initialized  to  be  sets  containing  the  single  set  of  items  To,  the 
closure  of 

{Do  -  oiH) 

Repeating  until  there  are  no  more  sets  of  items  in  C' , 

(2)  A  set  of  items  /  is  removed  from  C' . 

(3)  For  each  vocabulary  symbol  X  in  G,  the  set  of  items  generated  from 
GOTO(/,A')  is  added  to  C  and  C' ,  unless  it  already  exists  in  C. 

Once  completed,  the  sets  of  items  C  returned  by  ITEMS  corresponds  to  the  states 
of  a  DFA  recognizing  the  viable  prefixes  of  grammar  G',  while  the  GOTO  function 
for  this  set  of  items,  referred  from  here  on  as  GOTOg,  corresponds  to  the  transition 
function  of  that  DFA,  mapping  state  to  states. 

The  functions  CLOSURE,  GOTO,  and  ITEMS  are  defined  in  Figure  B.l.  The  canonical 
set  of  items  generated  for  the  grammar  of  Figure  5.1  is  shown  in  Figure  B.2.  The  GOTO 
function  for  this  set  of  items  is  shown  as  a  transition  diagram  of  a  DFA  in  Figure  B.3. 

ITEMS  (G') 

Let  C  :=  {CLOSURE({D0  —  oRd})} 

For  each  /  €  C, 

for  each  distinct  X  |  A  — *  aoX0  £  I , 
let  C  :=  C  U  {GOTO  (/.A)} 

Return (C) 

CLOSUREC/) 

For  each  (p,j)  £  I  |  j  p, 

^Vo+i)  ^  ^  • 

for  each  Dq  £  N  \  Gp(;+1)  =  Dq, 

let  I  :=  I  U  {(q, 0)} 

Return(/) 

GOTO  (/.A') 

CLOSURE (U^—ao A' y?€/  A  -  aXo/3 

Fig.  B.l — Constructing  the  canonical  set  of  LR(0)  items 
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lo¬ 

D0  — ’  oS  H 

h- 

NP  —  *detc*n 

S  —  oNP  VP 

S  —  oS  PP 

/f,: 

NP  —  +det  *no 

NP  — *  o*n 

NP  — *  c*det  *n 

h- 

NP  —  *no 

NP  —  oNP  PP 

S  —  S  PPo 

ll  ■ 

Do  —*  SoH 

S  —  SoPP 

h: 

S  —  NPcVP 

PP  —  o*prep  NP 

NP  —  NPoPP 

VP  —  o*v  NP 

face  '■ 

Do  — *  S  Ho 

PP  —  o*prep  NP 

I 2  : 

PP  — >  *prepoNP 

VP  —  *voNP 

NP  — -  o*n 

NP  — -  o*n 

NP  — >•  o*det  *n 

NP  —  o*det  *n 

NP  —  oNP  PP 

NP  —  oNP  PP 

h- 

PP  — *  *prep  NPo 

hr- 

VP  — *  *v  NPo 

NP  —  NPoPP 

NP  -*  NPoPP 

PP  — <•  o+prep  NP 

PP  -+  o*prep  NP 

I 4  : 

NP  —  NP  PPo 

- 

S  —  NP  VPo 

Fig.  B.2 — Canonical  LR(0)  collection 


Fig.  B.3 — GOTOc  graph 
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Constructing  the  Extended  LR  Parse  Table 

Given  the  canonical  set  of  items  C  denoting  a  DFA  that  recognizes  viable  prefixes  of 
G,  we  next  construct  an  extended  (multi-action)  LR  parse  table  from  C  in  terms  of  the 
ACTIONS  and  GOTO  functions  called  by  the  parser.  These  functions  are  constructed  using 
a  modification  of  the  SLR  parse  table  construction  technique,  as  described  below. 

Definition.  To  determine  reduce  actions,  the  construction  algorithm  requires  the  func¬ 
tions  FIRST  and  FOLLOW. 

•  FIRST(a)  returns  the  set  of  terminals  and  preterminals  that  can  begin  a  string 
derived  from  a.  If  a  ^  c,  then  c  is  also  returned  by  FIRST(a). 

To  compute  FIRST)  A)  for  all  vocabulary  symbols  A  (i.e.,  strings  a  of  length  I), 
apply  the  following  rules  until  nothing  more  can  be  added  to  any  FIRST(A)  set. 

(1)  If  AT  is  a  terminal  or  preterminal,  then  FIRST(A')  is  {A"}. 

(2)  If  X  is  a  nonterminal,  then  for  each  production  A'  Y a  such  that  Y  is  a 
terminal  or  preterminal,  add  Y  to  FIRST(A').  If  A”  — <•  e  is  a  production,  add  e 
to  FIRST(A). 

(3)  In  general,  if  X  is  a  nonterminal,  then  for  each  production 

X  —*  Cpi  Cp2  •  •  ■  Cpp 

and  some  i  such  that  all  of  Cp\,  ■  •  •  ,  Cp;_i  are  nonterminals  and  FIRST(CW) 
(for  j  =  1,  •••,*  -  1)  contains  e  (i.e.,  Cp\,  •  ■  ■  ,  Cpi_i  e),  add  every  non-e 
symbol  in  FIRST(Cp*)  (for  *  =  1,  •  •  -,?)  to  FIRSTLY).  If  e  is  in  FIRST(Gpj ) 
for  all  j  =  1,  •••,!>,  then  add  e  to  FIRST) A'). 

Similarly,  to  compute  FIRST(o)  for  any  string  of  vocabulary  symbols  a  =  A'i  •  •  •  A"n 
apply  the  rules: 

)4)  If  ATj  is  a  terminal  or  preterminal,  then  FIRST(A'i  •  •  •  A'n)  is  {A]}; 

(5)  For  some  i  such  that  Aq,  •  •  •,  A',_j  are  nonterminals  and  FIRST(A'j)  (for  j  = 
1,  •  ■  • ,  A'i_i )  contains  e,  add  every  non-c  symbol  in  FIRST) A'*)  (for  k  -  1.  ■  •  •,  i) 
to  FIRST) Xi  ■  ■  -  A'n).  If  c  is  in  FIRST) Xj )  for  all  j  =  1 ,  then  add  f  to 
FIRST) Ai  ••■A „). 

•  FOLLOW(A)  returns  the  set  of  terminals  and  preterminals  that  can  appear  imme¬ 
diately  to  the  right  of  nonterminal  A  in  some  sentential  form  of  G'.  That  is,  the  set 
of  terminals  and  preterminals 

{A  |  D0  a  A  A'/!} 

If  A  can  be  the  rightmost  symbol  in  some  sentential  form  of  G,  then  the  cnd-of- 
sentence  symbol  ‘T  appears  in  FOLLOW) A). 
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To  compute  FOLLOW(A)  for  all  nonterminals  A,  apply  the  following  rules  until 
nothing  can  be  added  to  any  FOLLOW(A)  set. 

(1)  The  end-of-sentence  symbol  ‘T  is  in  FOLLOW'!/?),  where  R  is  the  root  of  CL 

(2)  For  each  production  A  — •  aB.3,  such  that  3  is  not  empty,  add  everything  in 
FIRST(/?),  except  c,  to  FOLLOW(B). 

(3)  For  each  production  A  —  oB/j,  such  that  3  is  empty  or  FIRST(/3)  contains  c, 
add  everything  in  FOLLOW(A)  to  FOLLOW(B). 

The  FIRST  and  FOLLOW  tables  computed  for  the  grammar  in  Figure  5.1  are  shown  in 
Figure  B.4. 

Notation.  Let  C  =  {/q,  7i,  •  •  ■ ,  In}  be  the  canonical  set  ofLR(O)  items  constructed  for 
grammar  G\  and  let  GOTO<y(/,Ar)  be  the  associated  transition  function.  The  states  of  the 
parse  table  are  Sq<  S\ ,  •  •  • ,  Sn,  where  state  5,  is  constructed  from  /,. 

The  Constructor.  The  constructor  is  a  function  of  one  argument  C'ONST(C).  CONST 
determines  the  entries  of  the  action  and  goto  tables  for  each  state  5,  using  the  following 
rules. 

(1)  For  all  items  Dp  — »  aoX/3  in  /,  such  that  GOTO(y(/;,.Y)  =  lj,  if  X  is  a  terminal 
or  preterminal,  add  shj  to  ACTIONS!  St,  X ),  otherwise  set  GOTO(5,,A')  to  j. 

(2)  For  all  items  Dp  — >  qo  in  /,,  where  p  ^  0,  add  rep  to  ACTIONS!  Si, AT)  for  all  „Y 
in  FOLLOW(Dp). 

(3)  If  item  Do  — i ►  R  H°  is  in  then  add  acc  to  ACTIONS(5,,d). 

(4)  All  entries  not  defined  by  rules  (1)  through  (3)  are  made  errors. 

The  action  and  goto  tables  of  the  grammar  in  Figure  5.1  appear  in  Figure  5.2. 

A  Space- Efficient  Implementation  of  the  Constructor 

A  space-efficient  algorithm  for  constructing  an  extended  LR  parse  table  appears  in 
Figure  B.5.  Figure  B.6  defines  the  ACTIONS  and  GOTO  functions  of  the  parser  in  terms 
of  this  table  representation. 


X 

FIRST!*) 

FOLLOW(Ar) 

s 

{*det,*n} 

{-3,+prep} 

NP 

{*det,*n} 

{H,*prep,*v} 

VP 

{*v} 

{H,*prep} 

PP 

{♦prep} 

{H,*prep,*v) 

Fig.  B.4— FIRST  and  FOLLOW  tables 
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in  CONST,  [1]  initializes  C.  the  set  of  item  set/state  pairs,  and  then  constructs  the 
start  state  corresponding  to  the  initial  item  (0,0). 

In  GET-STATE,  [2]  checks  if  a  state  of  item  set  /  has  not  already  been  recorded  in  C, 
returning  that  state  if  it  has  and  constructing  that  state  otherwise. 

In  CONST-STATE,  [3]  initializes  the  action  and  goto  tables  of  a  new  state  S  to  be 
associated  with  I  in  C;  T'  is  a  temporary  table  used  to  record  shift  and  goto  actions;  [4] 
iterates  over  each  item  in  the  closure  of  /.  If  an  item  indicates  the  complete  recognition 
of  production  p ,  [5]  enters  a  reduce  action  in  the  action  table  under  each  terminal  and 
preterminal  that  can  follow  nonterminal  Dp\  otherwise,  there  will  be  a  transition  across  the 
j  +  1st  symbol  of  production  p  into  the  item  set  V .  This  transition  will  result  in  a  shift 
action  or  a  goto  and  [6]  records  it  in  the  temporary  transition  table  T' .  After  recording  all 
reduce  actions  and  noting  all  transitions,  [7]  iterates  through  the  transitions  in  T' .  For  each 
transition  into  the  item  set  [8]  accesses  the  state  S'  associated  with  /'.  If  the  transition 
is  across  a  nonterminal,  [9]  makes  the  appropriate  entry  into  the  goto  table;  otherwise,  [10] 
adds  a  shift  action  to  the  reductions  already  recorded  in  the  action  table;  finally,  [1 1]  returns 
the  newly  constructed  state. 
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CONST  (G') 

[1]  Let  C  be  empty 

Return (CONST-STATE ({(0,0)})) 

GET-STATE(Z) 

[2]  If  3(1,  s)  €  C, 

return (s) 
else 

return (CONST-STATE (/) ) 

CONST-STATE (/) 

[3]  Let  Tactlon  ,  Tgoto  and  T1  be  empty 
Let  s  (TaCfton,  Tg0to) 

Let  C  :=  C  U  {(/,s)} 

[4]  For  each  (p,j)  €  CLOSURE (/) , 

if  j  —  'p 

for  each  X  €  FDLL0W(Dp) 
if  3(X,  a)  6  Tactl0n  > 
let  a  :=  a  U  {‘rep’} 
else 

let  Taction  :=  Taction  U  {(A,{  re  p’})} 

else 

[6]  if  3 (CpU+i),l>)  e  r 

let  I'  :  =  /'  U  {( p,j  +  1)} 

q2_  g  Q 

let  T1  :=  T1  U  {(Cp(J+1),  {(p,j  +  1)})} , 

[7]  For  each  (X,!')  €  T’ , 

[8]  let  s'  :=  GET-STATE (/') 

[9]  if  X  €  N, 

let  1  goto  :=  Tgoto  U  {(^\5)} 

else 

[10]  if  3 (A'  ,  fl)  €  Taction  » 

let  a  a  U  {‘sh  s'’} 
else 

let  Taction  :  =  Tactl0n  U  { (X,  {  ‘  sh  s' '  }) } 

[11]  Return(s) 

Fig.  B.5 — The  constructor 


ACTIONS  (s  =  (Taction,  Tgoto)  ,x) 

If  3(x,o)  €  Tactl0n , 

return(a) 

else 

return ({  }) 

G0T0(s  =  (Taclton,  Tg0t0), Dp) 

ReturnCs'’)  |  ( Dp,s ')  e  Tgoto 

Fig.  B.6 — The  ACTION  and  GOTO  functions 
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